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ABSTRACT 

The previous history of unsystematic practice of 
educational evaluation has created for the present evaluator a set of 
problems resulting from a tradition of fear, mistrust, suspicion and 
disbelief among the educational community. Absences of evidence of 
critical influence have resulted in the underfunding of evaluation, 
common problems faced by evaluators can be attributed to the current 
state o^ the art and to the present status of education, social 
action programs have three major characteristics; (1) The goals or 
intentions of the programs seek to effect changes in society as a 
whole; (2) The programs are outgrowths of social theory and due to 
the changing nature of social theory often fail to be accompanied by 
operational prototypes; and (3) The programs are action- oriented and 
fail to provide time for indecision. Problems to be dealt with by 
evaluators of social action programs arise from the uniqueness of the 
program goals or from the developmental nature of the programs. 
Problems faced include: (1) the presence of value conflicts and the 
absence of procedures for negotiation, (2) inability of decision 
maker to identify total information needs, (3) unrealistic schedules 
or goals, (4) length of time required for completion of programs, (5) 
measurement of uncharted relationships, and (6) inability to analyze 
complex interactions. (CK) 
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The pievious history of unsystematic practice of educational evaluation has 
created for the present evaluator a set of prcblems resulting from a tradit.ca 
of fear» mistrust* suspicion and disbelief among the educational community. Ed~ 
ucational management often fails to be as systematic and scientific as it is hy*~ 
potheslzed to be; descriptions of program objectives and activities often fail to 
serve as unique identifiers which adequately communicate to non~program personnel 
program similarities and differences. The present status of measurement in edu- 
cation is lacking the skills to deal adequately with several variables of major 
concern* such as consistencies of philosophy and practice. Current design and 
analysis models fail to accommodate the complexity of the educational environment 
and fall to produce information upon critical re] tlonshlps. Methods of report- 
ing results and of juslng data lack the simplicity and clarity often demanded in 
their application. Absences of evidence of critical Influence have resulted in 
the underfunding of evaluation. Efforts to defend and to Interest educators in 
further evaluation commitments have often resulted in educational administrators' 
expecting too much* too soon* of evaluation. 

Hence* there exists an array of ydommon problems faced by evaluators which 
can be attributed to the current state of the art and to the present status of 



' education. These problems are generally faced by all evaluators* regardless of 



ever* a group of problems not usually mentioned as common problems that are more 
attributable to specific evaluation tasks. These problems usually differ accord* 
ing to the types of programs being evaluated. This paper will be concerned with 
dealing with those problmns that are associated with social action programs. 

Social action programs for the most part can be thought of as_Jiavlng three 




the size* scope or nature of the enterprise being evaluated. There exists* how- 
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major characteristics: (1) the goals or intentions of the programs seek to effect 

changes in society as a whole; (2) the programs are outgrowths of social theory and 
'-^"due to the changing nature of social theory often fall to be accompanied by rpera- 
tlonal prototypes; and (3) the programs are action-oriented and fail to provide 
time for indecision. Many of the social action programs are based upon reform and 
tend to be directed toward areas where value controversies exist. 

Generally, the special problems to be dealt with by evaluators of social ac- 
tion programs arise from the uniqueness of the goals of the social action programs 
or from the general unchartedness or developmental nature of the programs in ques- 
tion. The problems that will be discussed in this paper generally do not have 
solutions. This lack of solution, however, does not prevent them from being sig- 
nificant, critical barriers in the design of evaluation and worthy of discussion. 

The first problem to be faced by the evaluator of social action programs 
deals with the presence of value conflicts and the absence of pr oced ures through 
which these value conflicts can be negntiated. Social action programs usually 
maintain reform as a major focus. In cases of social reform, value conflicts ex- 
ist as a given. Reformers usually become involved in activities dealing with re- 
form through dedication. Social action programs as a whole tend to attract dedi- 
cated people; yet, as in so many other programs, those attracted fall to have the 
same set of goals. 

s 

A first component of this problem is that the nature of the value conflict 
encountered in social action programs is of such an Intensity that Interpersonal 
relationships among leaders and administrators often break down, whereas in other 
education programs conflicts regarding values or outcomes are visually superseded 
by overriding common goals or objectives. 

A second component of the problem involved with the presence of value con- 
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fllcts Is that the evaluator becomes the scapegoat for the animosities generated 
by the value conflict » especially as he attempts to negotiate these conflicts. 

Social action pro«;rdms usually require of the evaluator a rather close and con- 
tinuing association with program management. During this continuing association, 
the evaluator must be very careful to receive adequate objective commentary from 
people holding controversial points of view, such that an objective treatment of 
the evaluation activities can be operationalized. 

There is also the problem that the evaluator, being surrounded by dedicated 
people actively Involved In a process of change, can either act as a reactive 
agent hindering the change or place himself into the program as part of the pro- 
gram, catching himself up In the change process and setting his objectivity at a 
less than desirable level. Resolution of value conflicts Is not absolutely essen- ^ 

tlal for good evaluation of social action programs, but may be necessary In the 
interpretation of results which, as one can see later In other discussion, are 
scarcely predetermined criterion observations. Hence, the absence of either a 

“ \ 

clear role definition or of procedures or checkpoints that an evaluator can follow 
in dealing with value conflicts presents the evaluator with a serious concern in 
social action programs. This concern probably is true of other programs, but not 
to the dimensions that the social action-type programs usually offer. 

The second concern deals with the ability of a jd^rl slon mnkpr^o Identify 
Important baseline information needs at the pnset_o£--ardevelopmental program. 

Social action programs generally tend to be developmental In nature In that they 
are seeking to right particular social injustices or to carry on social reform at 
a particular period In time. Associated with the development of social action 
programs Is the basic need to develop treatments and activities aimed at the ac- 
compllsYiment of a set of outcomes that may be very reactive to the setting In 
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vhich they take place. Hence, social action programs tend to be uncharted trips 
into the unknown. Current limitations of educational measurement require the 
establishment of certain baseline activities prior to the collection of informa- 
tion upon change. Such baseline activities include pretesting, development of 
tests, and specification of test development and storage instruments. During the 
time that the evaluator of a social action program is identifying the variables 
which will provide information of importance to the program objectives, it is 
assumed that there exists certain pre-knowledge concerning what is Important to 
be measured. Unfortunately, in most social action programs the questions that a 
decision maker might ask, the decisions that he might have to make and which he 
might have to establish are not clear nor easy to anticipate. Hence, the deci- 
sion maker cannot totally identify the information needs which he may be expected 
to meet. The nature of social action programs (i.e., the fact that their objec- 
tives are based upon what are often value controversies) is such that support in- 
formation when needed is often imperative. Absence of particular support infor- 
mation at times *can be construed by program opponents as inexcusable and can be 
used to threaten program goal accomplishment. It becomes more imperative, then, 
for the evaluator to collect an adequate baseline of information, but the clrcum- 
stances and lack of previous information on the variables involved in social 
action programs limit our ability to anticipate those information needs, creating 
a crucial situation. 

The third problem is related to, the reasonablm^s of the goals of social 
action progr^s. Too often, social action program goals posit either an unreal- 
istic sdiedule of attainment or a stated set of goals that cannot be logically 
attained in the future. In cases where the social action programs assume goals 
beyond our capabilities in terms of time periods for attainment, they usually 
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operate under an assumption that educational activity or education as a profession 
has the benefit of knowledge that is currently not In existence. Hence, social 
action goals oftejL assume that things like attitude measurement can occur without 
Instrument development or that attitudes can be changed In one-*day intensive set* 
tlxigs of operations, or that proper educational activities can be prescribed im- 
mediately to remedy diagnosed educatlona!! difficulties. Unfortunately, these 
assumptions of greater knowledge and greater power of education ultimately result 
in goals far too ambitious to be accomplished by the program* Some of these goals 
even go beyond the route of logic, mainly because the actual problem has not real** 
ly been analyzed; but Instead, focus has been placed upon the Injustice of the 
problem. A good example of this exists in Title I of the Elementary and Secondary 
Education Act; Title 1 was directed toward the equalization of educational oppor** ^ 

tunltles and focuses upon target groups of children in schools in areas where 
large numbers of poverty families are compacted. Educator-operationalized goals 
of Title I end up seeking to see if Title 1 students approach au equality 7lth ^ 

groups of similar age students In non-Tltle I schools in regard to achievement. 

It is quite clear here that Title I students are already learning at a less than 
average rate, and that the treatment of Title I is expected to Increase their rate 
of learning up to that of the average child. It is not impossible but may be 
overly ambitious to expect that Title I as a program might equalize the learning 
rate of the disadvantaged child versus the normal child in the non-Tltle I school; 
it is, however, ridiculous to expect that such a treatment could be devised that 
would not only equalize the learning rate but cause the already disadvantaged 
Title 1 child to learn at a faster rate than normal. Such an assumption would 
have to occur if the treatment were to equalize the children* s achievement status. 

A fourth problem is again associated with that of goals. This problem deals 
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with the fact that principal outcomes of social action programs are not usually 
observable until the operation of the program has become history. Examples of 
this center around such things as changes in social studies curricula » the pii~ 
mary effects of which become observable only after students have left the school 
setting. Similar social action goals are associated with programs like New Ca- 
reers, which seeks to cause a change in the characteristics of individuals fit- 
ting certain career opportunity program dimensions. However, until successful 
charting and practice of those programs are available and until sufficient time 
has passed to look at the effects of this program on the job market and on the 
status of certain minority groups, little can be said about the ultimate success 
of the career opportunities program. Instead, the evaluator is faced with meas- 
uring some short-term symptoms of what is hypothesized to be the long-term effect. / 

Hence, the evaluation of social action programs can only be as good as we can 
associate the short-term symptoms with the anticipated long-term effects. 

A fifth area of concern is related to the problem we have-with the measure- ^ 



ment of unanticipated, uncharted relationships. In social action programs we do 
not know the length of the critical time period upon which a treatment operates. 

In such examples as changing attitudes we do know that early measurement (l.e., 
measurement before the attitude* has taken place, anticipating quicker treatment 
reactions than actually occur) tends to cause false decisions to be made relative 
to the effectiveness of the program and tends to be reactive with the treatment. 

We also know that over-application of a treatment may eventiially result in just 
the opposite effect than that desired; to Illustrate this one might consider the 
conduct of an information band for attitude change. If the treatment is conducted 
for a length of time after the attitude has been changed, a resulting change may 
occur In the opposite direction. On the other hand, if the treatment is not con- 
ducted long enough to acquire the attitude change in the desired direction, this 
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change may never occur* What we do not know is the limits of that period over 
which the treatment can be applied and still output the desired change. In other 
words, we do not know the interval over which the attitude measurement should oc- 
cur. Many variables associated with social action programming have these charac- 
teristics — that they can be threatened by measurement made too early or by meas- 
urement too long delayed. Unfortunately, In many cases of social action program 
variables we do not know the critical period of measurement. 

A sixth and last problem is our inability to analyze complex interactions 
often appearing In social action programs. These interactions sometimes occur 
as part of the nature of the variables appearing within the social action program 
setting and at ether times occur as a result of the setting Itself. In the South 
it can be documented that blades as a group can be hated by a prejudiced farmer, 
yet that farmer feels only love and acceptance of his black farm workers. Like 
this seemingly paradoxical nature of prejudice, many social action variables 
behave with differences toward group norms and individual norms. Another inter- 
action problem occurs because we are unable to discern the effects of a treatment 
rendered individually versus the effects of a treatment rendered through mass 
media. In this case we are again unable to trace through a multivariate maze of 
subject type indicators to identify the outcomes of any specific subgroup under 
a particular admlnlstraclon of the treatment. Our inability to trace systemati- 

a 

cally results of highly specific target groups renders many of our analyses inad- 
equate to cope with the complexities of the social action programs. Our models 
for analysis simply do not cope with the ccmplexlty of the environment, and our 
previous information has not recorded for us an array of information concerning 
the Interactions of treatment with contextual variables. Hence, given that it is 
known what information is wanted and that data which could produce such information 
can be collected does not guarantee the existence of analysis models which would 
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allow such Information to be extracted. 

In summary, six problems have been Identified as seemingly peculiar, or at 
least unique to a point of concern, in social action programs. The author assumes 
that problems related to the specificity of other programs can be found through 
similar types of analysis and experience. Nowhere are suggestions made in this 
paper to satisfy some of these probl^s; instead, the author has brought these 
problems to the attention of the panel for discussion. Potentially, ^ch concerns 
may lead to the development of solutions and eventually to the further development 
of evaluation methodology. 
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